Repository navigation
feat(transport): add opt-in bounded upstream retry advice - #27
Conversation
UPSTREAM_RETRY_AFTER_ADVICE_ENABLED (default off) parses Retry-After and documented reset headers into bounded local advice. An allowed managed retry then waits for the longer of the existing backoff and the advice, and is skipped when the wait would pass the request deadline. Parsing never grants a retry, and upstream headers are never rewritten. The managed retry scheduling moves into services/managed_dispatch.py and behaves exactly as before while the flag is off.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
multillm-proxy | aa62d68 | Oct 08 2026, 09:47 PM |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aa62d6816d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def retry_seconds(headers, now: float, default: int = 60) -> int: | ||
| settings = retry_advice_settings() | ||
| if settings.enabled: | ||
| advice = parse_retry_advice(headers, now=now, max_seconds=settings.max_seconds) |
There was a problem hiding this comment.
Pass the actual upstream status into advice parsing
When advice is enabled, this call omits status_code, so parse_retry_advice classifies every free-route outcome as its default 429. The callers in routes/free_routes.py also use retry_seconds for 401/403/404 and other failures; consequently, a Retry-After on an authentication or caller error is accepted and can block a provider or model for up to the configured maximum, even though the parser explicitly rejects advice for those statuses. Add a status parameter and pass the available upstream status so these responses use the fallback cooldown instead.
Useful? React with 👍 / 👎.
30f8c89
into
integrate/gateway-waves-1-2
Summary
Adds bounded parsing of upstream retry and reset hints behind
UPSTREAM_RETRY_AFTER_ADVICE_ENABLED, which defaults to false. While the flag is off, retries, waits, headers, bodies and free-pool cooldown timing are unchanged.services/retry_advice.pyparsesRetry-After(seconds or HTTP-date),X-RateLimit-Reset,X-RateLimit-Reset-Requests/-Tokensand the Anthropic reset headers into immutable local advice. Malformed, stale and non-finite values are ignored.UPSTREAM_RETRY_AFTER_MAX_SECONDS(default 3600, at most 7 days). Upstream headers are never rewritten.services/managed_dispatch.pyholds the managed retry scheduling thatproxy_service.pypreviously repeated three times:Idempotency-Key.retry_secondsuses the same bounded parser while enabled.Details are in
docs/retry-advice.md.Shared registrations
UPSTREAM_RETRY_AFTER_ADVICE_ENABLEDandUPSTREAM_RETRY_AFTER_MAX_SECONDSto the Container env keys.Tests
tests/test_retry_advice.py: new. It includes registered-path tests for raw NanoGPT errors and Groq retries with the flag off and on.npm run test:worker: 1,227 passed.cf:migrations:check,test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.This is part of Wave 3. It is based on the Wave 2 integration branch (#25), and the Wave 3 integration PR adds the README line.